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ABSTRACT 

This final report summarizes work done to formulate 
and test some cognit ively-based principles useful for designing 
instruction in scientific or engineering fields. These principles 
stress the coherence and interpretability of the knowledge acquired 
by students. The principles were specifically applied to: (1) analyze 
the underlying knowledge and thought processes needed to interpret 
scientific concepts effectively; (2) investigate how actual experts 
and novice students interpret such concepts; and (3) devise and test 
instruction designed to teach such concepts more eff'^ctively . 
Attempts were also made to devise computational en^^^ironments to 
explore and implement these instructional ideas. (Aui.ior/RH) 
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Abstract 

This final report summarizes briefly work done to formulate and test some 
cognitively-based principles useful for designing instruction in scientific or 
engineering fields. These principles stress particularly the coherence and 
interpretability of the knowledge acquired by students. The principles were 
specifically .applied to analyze the underlying knowledge and thought 
processes needed to interpret scientific concepts effectively; to investigate how 
actual experts and novice students interpret such concepts; and to devise and 
test instruction designed to teach such concepts more effectively. Attempts 
were also made to devise computational environments to explore and 
implement these instnjctional ideas. 
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Introduction 

The goal of this project was to formulate and test some cognitively-based 
principles for designing instruction in complex domains, such as science or 
engineering. 

These instmctional principles were to be based upon the following: (a) An 
analysis of the underlying knowledge and thought processes needed for good 
performance in the particular domain of interest, (b) An adequate 
understanding of the knowledge and thought processes of students coming to 
the learning situation. The instructional process would then be explicitly 
designed to transform a student's knowledge and thinking so as to approximate 
those needed for good performance. 

To implement the project, it was planned to focus on a subject-matter 
domain which would be sufficiently limited In scope to be investigated in some 
detail, but which would be representative of the complexities encountered in 
broader realms of science or engineering. Accordingly, we chose to devote 
primary attention to the interpretation and learning of scientific or mathematical 
concepts (such as the physics concept "acceleration"). Indeed, an adequate 
understanding of such concepts is of fundamental importance to any scientific 
work and is an essential prerequisite for any problem solving in scientific 
domains. The leaning of such concepts presents many difficulties for students 
and conventional teaching methods are often quite ineffective. 

In carrying out our work, we were centrally interested in obtaining 
information sufficiently detailed to elucidate underlying thought processes and 
to assess the ways in which they would be affected by specific instructional 
interventions. Accordingly, we concentrated our efforts on detailed 
observations of individual subjects under controlled conditions. 



General Theoretical Considerations 

Analysis of desirable performance 

Our basic criterion for ultimately desired intellectual performance is that a 
person's knowledge should be effectively usable so that it can be applied 
correctly and flexibly in a wide range of situations. 

This performance criterion is far from trivial. Indeed, all too commonly the 
knowledge acquired by students as a result of ordinary instruction is merely 
"nominal"; i.e., it can be displayed by students on standard examination 
questions, but cannot be used flexibly by them to enhance their capabilities of 
independent thinking. 

The criterion of effective usability raises the following theoretical questions: 
What are some of the essential characteristics needed to make knowledge 
about some topic effectively usable? Our analysis sugges;ted that the following 
characteristics are of central importance: 
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(1) The topical knowledge must be accompanied by ancillary knowledge 
ensuring that the topical knowledge can be properly interpreted. Such 
interpretation knowledge must specify how to instantiate the topical knowledge 
in any particular case, and how to determine whether such instantiation is 
correct or not. (Such interpretation is ciearly an essential prerequisite for more 
complex problem solving.) 

(2) The form of the knowledge, i.e., its description and organization, is at 
least as important as its content to ensure that the knowledge is effectively 
usable. In particular, it is essential that the knowledge is adequately coherent , 
i.e., that knowledge 9lements are interconnected by possible (strict or plausible) 
inferences among them. Such coherence greatly facilitates ease of 
remembering, ease of regenerating knowledge that has been forgotten, ease of 
ensuring consistency, ease of debugging, and ease of extending the 
knowledge by generalization or new learning. 

(3) Available knowledge needs to be accompanied by adequate control 
knowledge, (a) Such control needs to ensure proper invocation, 
implementation, and checking when available knowledge is applied. (Indeed, 
particularly in science, disciplined knowledge application is a prerequisite for 
accurate and flexible knowledge use.) (b) Deliberate and automatic control of 
knowledge must be well-integrated. Indeed, performance of a complex task 
would be very difficult unless some lower-level knowledge can be used nearly 
automatically in ways that are fast and effortless (so as to leave adequate 
mental capacity available. for the performance of more complex aspects of the 
task). 



Instructional implications 

The preceding considerations lead to the following general instructional 
implications: 

(1) Instruction should be designed so as to explicitly ensure that any 
knowledge acquired by students have the preceding characteristics needed for 
good performance. 

(2) Learning can be faciliiated by maximizing these characteristics at any 
intermediate stage of the learning process. (For example, care devoted to 
keeping any newly acquired knowledge interpretable and coherent can 
appreciably facilitate extension of this knowledge by further learning.) 

The following paragraphs summarize briefly how we tried to implement and 
investigate tnese general theoretical guidelines in the specific domain of 
scientific concepts 
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Analysis of Concept Interpretation 

essential task in dealing with any scientific or mathematical concept 
(such as "acceleration", "area", ...) involves the ability to interpret this concept 
so as to instantiate it properly in any specific case. This interpretation task is 
actually more complex than one might naively suspect. 

To interpret such a concept one must retrieve from memory some knowledge 
stored about the concept, and then engage in information processing to apply 
this knowledge in the particular situation of i.nterest. Different possible modes of 
concept interpretation are then possible, depending on the kind of knowledge 
stored in memory and the kind of processing done with it. Each of those 
possible modes has some distinctive characteristics, in terms of unambiguity, 
precision, generality, and ease of use. Correspondingly, each of them can also 
lead to distinctive kinds of errors. 

We performed an analysis, buttressed by some observations, of various 
possible modes of concept interpretation and their implications. In brief, these 
modes of interpretation can be classified into the following two types: 

(1) Some interpretation modes are "formal", i.e., deliberately designed to 
achieve great precision and generality. They store in memory well-specified 
and general definitional knowledge about the concept, knovyledge which must 
then be explicitly processed to be applied in any specific instance. The 
knowledge stored may be a "declarative specification" of features characterizing 
the concept; the processing needed for application in particular instances may 
then require appreciable problem solving. Alternatively, it may be a "procedural 
specification" which provides a more detailed and explicit specification of how 
to interpret the concept in any particular instance. Such formal interpretation 
modes have the advantage of being reliably accurate, precise, and general. 
They have the disadvantage that their application is slow and requires 
significant mental effort. 

(2) Other interpretation modes are based on a repertoire of case-based 
knowledge.. The knowledge stored in memory consists then of knowledge 
about various special cases, standard cases, and typical kinds of cases of the 
concept. If this knowledge has become compiled as a result of familiarity and 
repeated use, such knowledge can then be invoked almost automatically by 
recognition processes that match this knowledge with any particular instance of 
interest. These non-formal case-based interpretation modes have the 
advantage of efficiency, i.o., they can be applied quickly and effortlessly. They 
have the disadvantage that they can easily lead to errors or inconsistencies, 
and are not well suited for making general inferences. 

What then is an "ideal" mode of concept interpretation which is both reliably 
effective and efficient? The preceding analysis suggests that such an ideal 
mode of concept interpretation should rely on both formal and case-based 
knowledge used in complementary ways. If one encounters a familiar situation, 
it is most efficient to use case-based knowledge, and then to rely on more 
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formal knowledge to check the resulting interpretation if necessary. On the 
other hand, if one encounters an unfamiliar situation, tries to detect or diagnose 
errors or inconsistencies, or needs to make general arguments, then it is most 
useful to rely on formal knowledge and to use case-based knowledge for 
possible check-points. 

An account of the preceding work was published as an ONR report (Reif, 
1986). An article based on this work has also been accepted for publication in 
the journal Cognitive Science (in press). 



Comparative Studies of Good and Poor Performance 

To obtain data bearing on the characteristics of usable knowledge, and 
more specifically on the preceding analysis of effective and efficient concept 
interpretation, we made some detailed observations to elucidate the nature of 
the underlying knowledge and thought processes used by persons who were 
either good or poor interpreters of scientific concepts. 

To this end, we (predominantly Lisa Quinn and I) constructed two carefully 
designed sets of questions requiring persons to interpret the concept 
"acceleration" qualitatively in a variety of situations. The persons used in this 
experimental investigation were either phyoics professors who had recently 
taught an introductory mechanics course dealing with the concept 
"acceleration", or students enrolled in such a course. Each such person was 
individually asked to talk out loud about his or her thinking while trying to 
answer the questions — once without interventions by the experimenter, and a 
second time with requests for fuller explanations of his or her thinking. The 
transcript of the person's tape-recorded utterances, together vith his or her 
written v/ork, constituted then a protocol which was subsequently analyzed in 
detail. 



Good experts 

Those experts (i.e., physics professors) who are good at interpreting the 
scientific concept seem to behave in a manner similar to that proposed by our 
ideal model of concept interpretation, (a) They can interpret reliably the 
knowledge invoked by them, (b) Their knowledge is quite coherent, i.e., 
consistent and usable for various inferences, (c) They have both formal and 
case-based knowledge, and they use both these kinds of knowledge in 
complementary fashion. 

However, e^/en these good experts (who answer correctly more than 90% of 
all questions given to them) occasionally mako mistakes. Many of these are 
due to the experts' inadequate control of their available knowledge (e.g. to 
failure to invoke appropriate knowledge, even when it was recently invoked In a 
slightly different context, or failure to check adequately application of this 
knowledge). 
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Novice students 



Even when students are currently enrolled in a course where they have 
studied and repeatedly applied the concept "acceleration" for more than a 
couple of months, their ability to interpret this concept is quite poor. In our 
investigation, these students were unable to answer correctly about 60% of the 
qualitative questions about this concept. 

Analysis of the protocols reveals that students' underlying knowledge about 
this concept has the following major characteristics: 

(1) Their knowledge is quite incoherent, consisting largely of various 
disconnected knowledge fragments (many of them incorrect).. As a result, 
students are often unable to deal with situations deviating slightly from standard 
situations previously encountered by them; they fail to detect inconsistencies; 
and they encounter paradoxes which they cannot resolve. 

(2) Correct concept interpretation, relying on on many such fragmentary 
knowledge elements, would require that each such knowledge element be 
accompanied by specific applicability conditions. However, not surprisingly, 
students often either fail to store adequately such applicability conditions, or fail 
to retrieve them when invoking a knowledge element. 

(3) Students rarely invoke general formal knowledge and are often unable to 
articulate such knowledge. Furthermore, when they do invoke such knowledge, 
they are frequently unable to interpret it and thus cannot apply it successfully in 
particular instances. 

In short, students' knowledge seems to lack most of the characteristics of 
interpretability and coherence which we have identified as essential to the 
effective usability of knowledge. 



Nominal ver sus good experts 

There are "nominal" experts (so designated by criteria such as title, position, 
degrees, or other credentials) whose actual performance is not particularly 
good. Hence it is important to specify whether nominal criteria or actual 
performance data are used to select "experts" in any comparative study of 
experts or novices. For example, in our experiments we found "experts" (i.e., 
physics faculty members at a university") who were not good at interpreting an 
elementary concept, like "acceleration", which they themselves had recently 
taught.. Indeed, their performance was partly reminiscent of that of novices. 
Such observations have not only sociological, but also cognitive interest. In 
particular, one can try to trace the performance deficiencies of such 
experienced persons to specific deficiencies in the form of their underlying 
knowledge. 
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Some results of these comparative studies of good and poor performance 
have been summarized in a recently published paper (Reif, 1987). We hope to 
present a more extensive account in a future publication. 



instructional Studies 

The preceding theoretical ideas and observations about the essential 
characteristics of flexibly usable knowledge suggest some specific guidelines 
for the design of instmction. 

Working with myself and Lisa Quinn, Peter Labudde (a postdoctoral visitor 
from Switzerland) carried out an instmctional experiment to test some of these 
ideas. To be specific, we wanted to investigate the instructional efficacy of 
implementing merely the following two design guidelines based on our 
cognitive considerations: (a) Ensure explicitly the coherence and 
interpretability of newly acquired knowledge, (b) Ensure that this new 
knowledge is also coherent with students' preexisting knowledge, and can be 
used to debug this knowledge. 

To implement these guidelines for the teaching of a new concept 
("acceleration"), the instructional intervention did the following: (a) It taught 
students an explicit procedural specification of the concept acceleration and 
then gave students some practice in applying this procedure in several diverse 
situations. (Specification of the concept in terms of a procedure was designed to 
make the new knowledge interpretable. The consistent application of this 
single procedure to all cases was intended to make the new knowledge highly 
coherent ) (b) It then asked students to use this procedural specification to 
detect, diagnose, and correct errors previously committed by themselves or 
errors purportedly committed by others. (By having to use their newly acquired 
coherent knowledge to confront their own prior notions or other common 
misconceptions, stijdents were supposed to restructure their entire knowledge 
in coherent form. They were also thereby learning quality-control needed to 
detect and correct deficiencies in their new and preexisting knowledge.) 

The instructional intervention in this experiment lasted about half an hour 
and resulted in the following main outcomes: (a) Students' abilities to interpret 
the concept acceleration improved markedly, from about 40% correct 
interpretations in a pretest before the instruction, to about 95% correct in a 
posttest after the instruction, (b) In their attempts to interpret the concept 
acceleration, students invoked many incorrect knowledge elements before the 
instruction, but invoked essentially no incorrect knowledge elements after the 
instmction. (c) Students' interpretation of the concept after the instmction was 
both effective and efficient. In simple situations, they relied on compiled 
knowledge about special cases of the acceleration; but in more complex 
vSituations, they reverted to the procedural specification to interpret the concept, 
or to check tentative answers based on compiled knowledge. 
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In both its explicitness and emphasis on the form of the knowledge acquired 
by students, this instmctional intervention differs significantly from the common 
methods used to teach scientific concepts. The results of this instructional 
experiment were also certainly encouraging, although not definitive. In 
particular, this instmctional experiment incorporated only a few central design 
principles identified by our analysis; additional tests of instructional efficacy 
would also be desirable. 

An account of the preceding work on instruction was published as an ONR 
report (Labudde, Reif, & Quinn, 1987). An article based on this work has also 
been accepted for publication in the European Journal of Science Education (in 
press). 



Computer-implemented Instruction 

Advantages of using computers 

There are many reasons why the use of computers would be advantageous 
in formulating, testing, and implementing instructional design principles of the 
kind discussed in the preceding paragraphs. The reasons include the 
following: 

Research on instruction can be made more rigorous by translating 
instructional designs into the form of instructional computer programs, (a) Such 
programs serve to make more explicit the assumptions incorporated in an 
instructional model, (b) They allow the testing of such models under controlled 
conditions where the successive actions of the student and the computer tutor 
are well-specified and well-known. 

The computer has some unique advantages as a medium, (a) It can provide 
powerful graphical representations, including dynamic representations of 
physical phenomena and of the student's own thought processes. (For 
example, a dynamic graphical representation may be used to portray a 
student's own progress in carrying out a procedure.) (b) The computer can 
easily store and then redisplay a student's past work on an intellectual task. 
Thus it allows the student to reflect on his or her past thinking — and also more 
easily to detect, diagnose, and correct his or her previous deficiencies, (c) With 
carefu; design, computers can provide an environment where students can 
actively explore new concepts while being constrained to behave in disciplined 
ways. 

Finally, computers would be very valuable for delivering practical instruction. 
If students are to acquire scientific knowledge that is actually usable, it is 
imperative that they become actively engaged in the process of constructing 
their own knowledge. The computer can help by providing exploratory 
environments and by playing the role of private individual tutor. By contrast, 
most usual teaching environments fail partly because they put students in 
passive roles where they spend most of their time merely listening or reading. 
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Difficulties of computer implementation 

The previous appealing advantages of using computers are, however, not 
readily realizable because of several major difficulties — all of which became 
very evident in our work during this project. 

Some of these difficulties involve computer hardware. The Implementation 
of instructional designs, of the kind discussed in the previous paragraphs, 
require?*, computers which sufficiently powerful to handle text, graphics, and 
good student response judging. At the time when we first tried to use computers 
for this project, Apple Macintosh computers just became available and seemed 
barely adequate to satisfy our requirements. Only much later did we gain 
access to a few much more powerful Xerox-1 108 workstations. But the software 
available on relatively new machines is limited. Furthermore, one quickly 
discovers, to one's dismay, that even expensive and powerful machines have 
basic software that is often remarkably "buggy" — thus making any work very 
frustrating and progress slow. 

Even if powerful machines were readily available, with well-functioning 
basic software, there would still be the need for a good "authoring environment" 
facilitating the design and implementation of instruction. In particular, an 
instructional designer and producer must be able to direct his primary attention 
to fundamental pedagogical issues, without constantly needing to deal with 
details of low-level programming. The designer also needs to be able to easily 
edit and easily modify whatever he or she produces, preferably in an 
environment which is the same as that ultimately seen by the student (i.e., which 
is WYSIWYG, "what you see is what you get"). 

During the project we did not have available anything even remotely 
resembling such an authoring environment — and even now some such 
environments are only beginning to emerge. Hence we spent considerable 
time and programming effort trying to bring into existence something which 
might begin to serve our needs. 



"Draw-Ed" authoring environment 

David Oster, one of the programmers working on the project, responded to 
our needs and suggestions by constructing an authoring environment exploiting 
the Macintosh computer. 

This environment uses available Macintosh software, "MacDraw", as an 
editor which can construct and modify various kinds of objects, including text, 
graphics, and active elements (e.g., "buttons" which can be clicked by the 
mouse to produce various specified responses). In this way one can readily 
construct and edit "frames" (i.e., complex displays ultimately presented to a 
student) in a way where the author can see exactly what the student will see. 
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This editor is then coupled to a data-management program (implmented by 
Oster and called "DrawEd") which consists of the following components to carry 
out various functions: 

information-transfer between editor and DrawEd Using "Switcher (another 
standard Macintosh software), frames created in the editor can be readily 
transferred into DrawEd, or transferred back to the editor for easy modification. 

Data base. DrawEd stores all frames in convenient pictorial form. These 
picture frames can be easily accessed either by name, or by browsing through a 
picture file where any picture can be displayed (either full-size, half-size, or 
quarter-size) and then visually selected with the mouse. 

Interpreter of active elements. Active elements in the MacDraw editor are 
accompanied by some special symbols ar.d instructions. DrawEd interprets 
these symbols and instructions so that the active elements actually have the 
specified effects when selected (e.g., so that buttons selected by the mouse 
actually lead to the display of specified other frames or windows). 

Tutor . DrawEd has an authoring mode and a tutorial mode, and it is very 
easy to switch between these two. In the authoring mode, DrawEd acts like the 
data-manager just described. In the tutorial mode it acts as a tutor interacting 
with the student in the fashion specified by the instructional designer. Thus 
DrawEd can be used by the designer for authoring and modifying instructional 
programs; but it can then also be used by a student as a tutor providing 
instruction. 

Transfer to LISP machine^ . Finally. DrawEd can translate all the information 
contained in the frames (i.e., text, graphics, and instructions) into LISP and 
transfer this information electronically (via RS-232 interface) to a Xerox-1108 
LISP workstation. This transfer capability has the following advantages: If 
desired, the Macintosh can be exploited as a cheap and readily available 
computer acting as a kind of "scratch-pad" (or electronic story-board) where 
instructional designs can be laid out and perfected up to a certain point. Once 
this is done, the information may then be transferred electronically to the much 
more expensive LISP machine for further editing and refinements, so as to 
exploit the much greater capabilities of those machines. 



Final status o f computer i mplementation 

The Draw-Ed authoring system became barely functional only near the end 
of our ONR project. Except for some small-scale testing, the project was thus 
not able to exploit this system for significant instructional research. 
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Continuing Worl< 

Although the ONR project terminated last February, some of the lines of work 
initiated under this project are continuing under different auspices (no longer 
supported by the ONR). In particular, our current efforts focus more broadly on 
several cognitive issues involved in th-9 learning and teaching scientific 
concepts. This domain of investigation seems important not only in its own 
right, but is also a good prototype domain within which to explore instructional 
principles of much wider applicability. Since Xerox LISP workstaiions have 
become more available to us, we are also increasingly trying to exploit these 
machines for our instructional work and to create suitable instructional 
auth^^ring environments for these machines. 
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